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(57) Abstract: A method and system for generating an aggregate website search database index are provided. The method makes 
a request for a web page to a web site selected from one or more web sites and applies a signature schema that is maintained in a 
repository. The method uses the signature schema document that is predefined using a query language to provide instructions for 
applications to extract identified data from the requested web page. The method then creates an index using the signature schema. 
The extracted data and index arc then stored to an aggregate database, wherein the index defines the relationships between data 
extracted from the requested web page identified by the signature schema. 
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METHOD AND SYSTEM FOR GENERATING AN AGGREGATE WEBSITE SEARCH 
DATABASE USING SMART INDEXES FOR SEARCHING 

CROSS REFERENCE 

[0001] This application claims the benefit of the prior filing of U.S. Provisional Patent 
Application Serial No. 60/924503 filed May 17, 2007, the disclosure of which is 
incorporated herein by reference. 

COPYRIGHT 

[0002] A portion of the disclosure of this patent document contains material which is 
subject to copyright protection. The copyright owner has no objection to the facsimile 
reproduction by anyone of the patent document or patent disclosure, as it appears in the 
Patent and Trademark Office patent file or records, but otherwise reserves all copyright 
rights. 

FIELD 

[0003] The present application relates generally to telecommunications and more 
particularly to a system and method for generating an aggregate website search 
database using smart indexes for searching. 

BACKGROUND 

[0004] Web sites host and provide information using web pages that are communicated 
electronically via a telecommunications network. Accessing this information by some 
client computing devices can be challenging. Computing devices are becoming smaller 
and increasingly utilize wireless connectivity. Examples of such computing devices 
include portable computing devices that include wireless network browsing capability as 
well as telephony and personal information management capabilities. 

BRIEF DESCRIPTION OF THE DRAWINGS 

[0005] Figure 1 is schematic representation of a system for content navigation. 
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[0006] Figure 2 is a schematic representation of a wireless communication device from 
Figure 1 . 

[0007] Figure 3 illustrates a flow of interactions among components of the system of 
Figure 1 . 

[0008] Figure 4 is a schematic representation of an aggregate website search database 
for e-commerce. 

[0009] Figure 5 is a schematic representation of tables in an aggregate website search 
database for e-commerce. 

[0010] Figure 6 is a schematic representation of an aggregate website search database 
index. 

[0011] Figure 7 illustrates a method of creating the aggregate website search database 
index. 

[0012] Figure 8 illustrates a method of querying the aggregate website search database 
index. 

[0013] Figures 9A-9D and 10A-10D respectively illustrate representative web pages 
rendered on a first browser window and portions of said representative web pages 
transcoded and rendered on a second browser window in accordance with an 
embodiment. 

DETAILED DESCRIPTION OF THE EMBODIMENTS 

[0014] The smaller size of such client devices necessarily limits their display capabilities. 
Furthermore the wireless connections to such devices typically have less or more 
expensive bandwidth than corresponding wired connections. The Wireless Application 
Protocol ("WAP") was designed to address such issues, but WAP can still provide a very 
unsatisfactory experience or even completely ineffective experience, particularly where 
the small client device needs to effect a connection with web sites that host web pages 
that are directed to traditional full desktop browsers. In addition, the ability to access 
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data from multiple websites concurrently and extract relevant data in meaningful ways 
can be difficult and time consuming. 

[0015] Signature schema documents, may be pre-defined using a query language to 
provide instructions for application by an engine to extract data from web pages of 
respective web sites for storage to an aggregate database. For a particular web page, 
signature schema instructions identify a web page family for the web page and extract 
desired data from the web page in accordance with its web page family. The instructions 
use signatures previously identified within web pages of the same family to distinguish 
the web page family (e.g. in accordance with a shared template for each family) from 
others of the web site and to distinguish the desired data from other data for the web 
page family. A gateway server may receive data from a web site and apply signature 
schema instructions maintained in a repository coupled to the engine. Extracted data 
can be cached to a database coupled to the engine to facilitate querying of the data to 
enable aggregate results to be presented to a client machine (e.g. a wireless 
communication device). Smart indexes can be generated based upon data stored in 
the aggregate database to facilitate knowledge based queries as opposed to keyword 
search queries. 

[0016] In accordance with the present disclosure there is provided a method for 
generating an aggregate website search database index, the method comprising: 
sending a page request to a web site selected from one or more web sites; receiving the 
requested web page from the selected web site; retrieving signature schema associated 
with the selected web site; applying signature schema to the requested web page to 
extract data identified by the signature schema; creating an index using the signature 
schema defined for the extracted data for the one or more web sites; and storing 
extracted data and indexes to an aggregate database comprising data extracted from 
the one or more web sites wherein the indexes define relationships between data 
extracted from the requested web page identified by the signature schema. 

[0017] In accordance with the present disclosure there is provided a system for 
generating an aggregate website search database using smart indexes for searching, 
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the system comprising: at least one computing device comprising a processor and a 
memory coupled thereto, said memory storing instructions and data for configuring the 
processor to provide a engine to: send a page request to a web site selected from the 
one or more web sites; receive the requested web page from the selected web site; 
retrieve signature schema associated with the selected web site; apply signature 
schema to the requested web page to extract data identified by the signature schema; 
create an index using the signature schema defined for the extracted data for the one or 
more web sites; and store extracted data and indexes to an aggregate database 
comprising data extracted from the one or more web sites wherein the indexes define 
relationships between data extracted from the requested web page identified by the 
signature schema. 

[0018] In accordance with the present disclosure there is provided a computer program 
product storing computer readable instructions which when executed by a computer 
processor configure the processor for: sending a page request to a web site selected 
from one or more web sites; receiving the requested web page from the selected web 
site; retrieving signature schema associated with the selected web site; applying 
signature schema to the requested web page to extract data identified by the signature 
schema; creating an index using the signature schema defined for the extracted data for 
the one or more web sites; and storing extracted data and indexes to an aggregate 
database comprising data extracted from the one or more web sites wherein the 
indexes define relationships between data extracted from the requested web page 
identified by the signature schema. 

[0019] In accordance with the present disclosure there is provided a method for 
generating an aggregate website search database index, the method comprising: 
sending a page request to a web sites selected from the one or more web sites; 
receiving the requested web page from the web site; retrieving signature schema 
associated with the selected web site; applying signature schema to the requested web 
page to extract data identified by the signature schema; creating an index using the 
signature schema defined for the extracted data for the one or more web sites; storing 
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extracted data and indexes to an aggregate database comprising data extracted from 
the one or more web sites wherein the indexes define relationships between data 
extracted from the requested web page identified by the signature schema and wherein 
the signature schema comprise extensible Markup Language (XML) documents 
comprising query language for extracting data from the selected web page; receiving a 
search query from a client machine for data stored in the aggregate database; 
determining a database index associated with one or more parameters of the search 
query; generating a database query based upon the received search query using the 
determined index; retrieving data from the aggregate database based on the database 
query; and providing the retrieved data to the client machine. The client machine may 
be a wireless device with the aggregate database and indexes resident on the device. 

[0020] Referring now to Figure 1 , there is illustrated a system 100 for content navigation 
via a telecommunications network. In a present embodiment system 100 comprises 
one or more client computing devices in the form of client machines 102A and 102B 
(collectively 102), a web site (servers 106, 107 and 109 respectively host web sites 
104, 103 and 105 and a gateway and schema server 120. Machines 102 are 
respectively coupled to communicate with gateway and schema server 120 to obtain 
web pages (e.g. 110) transcoded from web sites 103, 104 and 105 and to access 
aggregate data from the web sites through web server 125 hosting web site 150. 

[0021] In the present embodiment, web sites 103, 104 and 105 host web sites which 
contain data that is to be aggregated into database 126. For example, web site 104 
comprises a web server 106 serving web pages (e.g. 110) defined from a plurality of 
web page family templates 108A-108D (collectively 108) and web page content 
(described further herein below) from data store 112. In the present embodiment of 
system 100, gateway and schema server 120 is coupled to a schema repository 124 
from which to obtain a signature schema 122 for a particular web site. Signature 
schema documents (e.g. 122) provide instructions and data with which an engine 140 of 
server 120 can extract data from web pages (e.g. 110) and transcode same to a target 
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format to provide transcoded web page data (e.g. 130 and 132) to the respective 
requesting client machines 102A and 102B as described more fully below. Gateway 
and schema server 120 may also be coupled to a database 126 for retrieving/storing 
data extracted from web sites in accordance with its operations. The database 126 may 
be a relational database for storing extracted data object and elements and their 
relationships from web sites in relation to the defined signature schema. The stored 
data can be accessed by a Structured Query Language (SQL) to retrieve desired data 
from database 126. Signature schemas for respective web sites may be defined (e.g. 
coded) using a computing device 128 as described herein below. A web server 125 is 
coupled to the aggregate web site database 126 to enable access to the aggregated 
web site database 126 data by a web site 150. The web server 125 can also provide a 
data collection engine 152, or web crawler, for sending requests to web sites 103, 104 
and 105 for desired page and provide content to schema engine 140 for processing. 

[0022] Representative client machines 102 include any type of computing or electronic 
device that can be used to communicate and interact with content available via web 
sites. Each of the client machines 102 may be operated by a respective user U (not 
shown). Interaction with a particular user includes presenting information on a client 
machine (e.g. by rendering on a display screen) as well as receiving input at a client 
machine (e.g. such as via a keyboard for transmitting to a web site). In the present 
embodiment, client machine 102A comprises a mobile electronic device with the 
combined functionality of a personal digital assistant, cell phone, email paging device, 
and a web-browser. Such a mobile electronic device may comprise a keyboard (or 
other input device(s)), a display screen, a speaker, (and other output device(s) (e.g. 
LEDs)) and a chassis for housing such components. The chassis may further houses 
one or more central processing units, volatile memory (e.g. random access memory), 
persistent memory (e.g. Flash read only memory) and network interfaces to allow client 
machine 102Ato communicate over the telecommunication network. 

[0023] Referring now to Figure 2, a schematic block diagram shows an exemplary client 
machine 102A in greater detail. It should be emphasized that the structure in Figure 2 is 
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purely exemplary, and contemplates a device that may be used for both wireless voice 
(e.g. telephony) and wireless data (e.g. email, web browsing, text) communications. 
Client machine 102A includes one or more input devices which in a present 
embodiment includes a keyboard and, typically, additional input buttons, collectively 
200, an optional pointing device 202 (e.g. a trackball or trackwheel) and a microphone 
204. Other input devices, such as a touch screen, and camera lens are also 
contemplated. Input from keyboard/buttons 200, pointing device 202 and microphone 
204 may be received at a processor 208. Processor 208 may be further operatively 
coupled with a non-volatile storage unit 212 (e.g. read only memory ("ROM"), Erasable 
Electronic Programmable Read Only Memory ("EEPROM"), or Flash Memory) and a 
volatile storage unit 216 (e.g. random access memory ("RAM") speaker 220, display 
screen 224 and lights one or more (LEDs 222). Processor 208 may be operatively 
coupled for network communications via a subsystem 226. Wireless communications 
are effective via at least one radio (e.g. 228) such as for Wi-Fi or cellular wireless 
communications. Client machine 102A also may be configured for wired 
communications such as via a USB or other port and for short range wireless 
communications such as via a Bluetooth® radio (all not shown). 

[0024] Programming instructions that implement the functional teachings of client 
machine 102A as described herein are typically maintained, persistently, in non-volatile 
storage unit 212 and used by processor 208 which makes appropriate utilization of 
volatile storage 216 during the execution of such programming instructions. Of 
particular note is that non-volatile storage unit 212 persistently maintains a web browser 
application 86 and, in the present embodiment, a native menu application 82, each of 
which can be executed on processor 208 making use of non-volatile storage 216 as 
appropriate. An operating system and various other applications (not shown) are 
maintained in non-volatile storage unit 212 according to the desired configuration and 
functioning of client machine 102A, one specific non-limiting example of which is a 
contact manager application (also known as an address book, not shown) which stores 
a list of contacts, addresses and phone numbers of interest to user U and allows user U 
to view, update, and delete those contacts, as well as providing user U an option to 
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initiate telecommunications (e.g. telephone, email, instant message (IM), short message 
service (SMS)) directly from that contact manager application. 

[0025] Native menu application 82 may be configured to provide menu choices to user 
U according to the particular application (or other context) that is being accessed. By 
way of example, while user U is activating the contact manager application, user U can 
activate menu application 82 to access one or more menu choices available that are 
respective to contact manger application 90. For example, menu choices may include 
options to invoke other applications (e.g. a mapping application to map a contact's 
address) or communication functions (e.g. call, SMS, IM, email, etc.) on the client 
machine 102A for a particular contact. Menu application 82 may be associated to a 
particular input button (e.g. one of buttons 200) and invoked to provide a contextual 
menu comprised of one or more menu choices that are reflective of the context in which 
the button 200 was selected. Note that the options in a contextual menu are stored 
within non-volatile storage 212 as being specifically associated with a respective 
application. Menu application 82 may be therefore configured to generate one or more 
different contextual menus that are reflective of the particular context in which the menu 
application 82 is invoked. For example, in an email application where an email is being 
composed, invoking menu application 82 would generate a contextual menu that 
included the options of sending the email, cancelling the email, adding addresses to the 
email, adding attachments, and the like. The contents for such a contextual menu 
would also be maintained in non-volatile storage 212. Other examples of contextual 
menus will occur to those of ordinary skill in the art. 

[0026] Figures 9A-9D and 10A-10D respectively illustrate representative web pages 
rendered on a first browser window and portions of a subset of data from said 
representative web pages transcoded and rendered on a second browser window in 
accordance with an embodiment. Figure 9A illustrates a representative home web page 
960A of an e-commerce web site (e.g. 104) in a browser window 950. Window 950 is 
illustrative of a rendering to a large size display device (e.g. desktop monitor). Web 
page 960A comprises, among other things, a menu portion 952 and a primary content 
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display portion 954, in the example, showing various advertisements 955 for products. 
Figure 10A illustrates the menu portion 852 extracted and transcoded and rendered as 
a web page on a second browser window 1050. Window 1050 is illustrative of a 
rendering to a small size display device (e.g. of a wireless mobile device). In addition to 
transcoding as a web page, menu portion 952 may be transcoded for menu application 
82 e.g. for invocation when browsing the site 104 as referenced further herein. 

[0027] Figure 9B illustrates an exemplary product web page 960B in window 950 
showing various product data (collectively 966) including image 966A, price 966, title 
966C and description 966D data that is transcoded and shown in window 1050 of 
Figure 10B. Also transcoded is the web page hierarchy list 968 showing where the page 
is on the web site. 

[0028] Figure 9C illustrates an exemplary product list web page 960C in window 950 
showing a list of products (collectively 970). A subset of the product data such as image 
970A, price 970B, and title 970C is transcoded and shown in window 1050 of Figure 
10C. Note that multiple pages 972 may be provided for the list 970. 

[0029] Figure 9D illustrates an exemplary account checkout web page 960D in window 
950 showing a login form 980 for receiving account login and password, which form is 
transcoded and shown in window 1050 of Figure 10D. Though not shown, other 
checkout pages (e.g. for payment or order confirmation, etc.), search pages, product 
and information pages may be similarly transcoded. 

[0030] Returning now to Figure 1, web servers 106, 107 or 109 and gateway and 
schema server 120 (which can, if desired, be implemented on a single server) can be 
based on any commonly available server environments or platforms including a module 
that houses one or more central processing units, volatile memory (e.g. random access 
memory), persistent memory (e.g. hard disk devices) and network interfaces to allow 
servers 106,107, 109 and 120 to communicate over the telecommunications network. 
Web servers hosts software applications comprising instructions and data for generating 
and serving web pages dynamically from the template families 108 and current 
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informational content therefore from data store 112. Load balancing, security/firewall, 
billing, account and other applications may also be present as is well-known in the art. 

[0031] Gateway and schema server 120 hosts software applications comprising 
instructions and data for proxying requests and responses between the client machines 
102 and web sites 103, 104 and 105. In addition to software for maintaining HTTP 
communications, performing requests, maintaining sessions, handling cookies, etc., 
engine 140 may be implemented in software to apply the signature schemas to web 
pages from web sites. There may be provided an interpreter that interprets the 
signature schema document and applies the actions against the web page code (as an 
ASCII (plain text) document)to extract desired data to produce a result set. A renderer 
may be provided to express the desired data result set (i.e. transcode to a target format 
such as cHTML (Compact HTML) for a mobile device browser) for transmitting to the 
client machines also in accordance with the signature schema. 

[0032] The web server 125 provides web pages to the requesting client machine 
through a browser or application on the client for rendering. The web data may be 
directly pushed to client machines 102A by e-mail or by other push based applications, 
or the data may be accessed by queries to web site 150 directly. The web site 150 may 
also extract content from the aggregate database 126 and apply a signature schema 
122 to the extracted database data, which schema may be configured to transcode the 
data in accordance with the target client machine 1 02A to tailor the output result. 

[0033] Machines 102, schema server 120 and web sites 103, 104, 105 and 125 are 
coupled via a telecommunication network (not shown) typically comprising one or more 
interconnected networks that may include wired and (at least for machine 102A) 
wireless networks. It should now be understood that the nature of the network is not 
particularly limited and is, in general, based on any combination of architectures that will 
support interactions between client machines 102 and servers 106 and 120. In a 
present embodiment the network includes the Internet as well as appropriate gateways 
and backhauls. 
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[0034] More specifically, in the present embodiment, a wireless network for client 
machine 102A may be based on core mobile network infrastructure (e.g. Global System 
for Mobile communications ("GSM"); Code Division Multiple Access ("CDMA"), 
Enhanced Data rates for GSM Evolution ("EDGE"), Evolution Data-Optimized ("EV- 
DO"), High Speed Downlink Packet Access ("HSPDA"), Universal Mobile 
Telecommunications System ("UMTS"), etc.) or on wireless local area network ("WLAN") 
infrastructures such as the Institute for Electrical and Electronic Engineers ("IEEE") 
802.11 Standard (and its variants) or Bluetooth or the like or hybrids thereof. In the 
present embodiment of system 100 it is contemplated that client machine 102B may be 
another type of client machine such as a PC (desktop or laptop) configured to include a 
full desktop computer or as a "thin-client" . Typically such have larger display 
monitors/screens than portable machines like 102A. A wired network for system 100 
and machine 102B can be based on a T1 , T3 or any other suitable wired connection. 

[0035] As previously stated in relation to Figures 1 and 2, each of the client machines 
102 is configured to interact with content available over the network, including web 
pages on web site 104. In a present embodiment, client machines 102A and 102B may 
navigate for content using a browser application (e.g. 86). As will be explained further 
below, on client machine 102A, browser application 86 may be a mini-browser in the 
sense that it may be configured to render web pages on the relatively small display 224 
of client machine 102A. Often, during such rendering, those pages are presented in a 
format that may be different from how those pages are rendered on a traditional desktop 
browser application (e.g. browser 86 of client machine 102B). Mini-browsers typically 
attempt to convey substantially the same information as if the web pages had been 
rendered on a full browser such as Internet Explorer®, Safari® or Firefox® on a 
traditional desktop or laptop computer like client machine 102B. 

[0036] Figure 3 is a flowchart illustrating operations/interactions for generating and 
maintaining an aggregate database from web sites 103-105 for populating and updating 
database 126, The flow chart provides an example of the interaction among the 
gateway and schema server 120/140, and data collection engine 152, with web 
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servers 106, 107 and 109 hosting web sites 103, 104 and 105 to generate and maintain 
the aggregate database 126. The data collection engine 152 (DCE), makes a request 
302 to the web site's web server (for example web server106) for the specified pages 
based upon the type of data to be aggregated. The web page code(e.g. 110) is 
generated by the server and sent 306 to DCE 152. The web page code received is a 
text file. It typically does not include objects referenced by the code such as images, 
video, audio, further web pages, etc. that are typically subsequently retrieved and 
inserted at the time of rendering a web page by a browser. The schema server engine 
140 (SSE) (for example, in parallel or without waiting for a response from server 106) 
makes a request 304 to the signature repository 124 for the signature schema 
document 122 for the web site, which request may use the domain in the URL as an 
identifier for obtaining the document 122. The schema server engine 140 receives 308 
the schema and does not render the web page 110 per se but instead uses the 
instructions in the signature schema document 122 to extract the desired data from the 
web page 110. The signature schema 122 is configured to extract data from web page 
110 in accordance with the specific desired content characteristics for the database 126 
which is based upon the target client machine 102A, and having knowledge of display 
224 capabilities - such as screen size, resolution, and other one or more parameters - 
useful in determining the way in which the data is to be displayed on the machine 102A. 
The web page 110 or extracted data is stored 310 in database 126 in a relational 
database structure, where data related to the defined signature scheme from the web 
site is stored. Indexes can be created to reference the stored data providing improved 
query ability. The creation indexes can be tailored based upon the signature schema 
associated with the web site page as the schema extracts objects and attributes with 
knowledge i.e. knows objects title/name, image, price, description relationships between 
objects can be developed. 

[0037] Client machine 102A can then make a request 312 to web site 150 on server 125 
for a query to the database 126 regarding desired web sites having a specific domain 
(URL). Web site 150 requests 314 relevant data from the database 126. The results 
are extracted 316 and are sent 318 to the client machine as aggregate results, or as a 
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proxy as if the query was made directly to the source web site, and transcoded in 
accordance with the schema 122, to the requesting client machine 102A processed by 
the signature schema engine 140 before presentation by web server 125. Alternatively, 
the data may be pushed to a client machine to a push based application. As noted 
above, retrieved data 130 may also comprise transcoded navigational data for menu 
application 82 and informational content data (e.g. a list of products and related 
information from a web page) for displaying by browser application 86. The process can 
then be repeated for each identified website such as web sites 103 and 105. 

[0038] Signature schemas are pre-defined documents, and may be extensible Markup 
Language (XML) documents utilizing an SQL-like query language, to incorporate 
instructions and data with which to intelligently extract the data from web pages (which 
web pages are typically coded in HTML, DHTML, XHTML, XML, RSS, Javascript, etc). 
This extracted data may be transcoded and provided to client machines 102, used to 
dynamically generate a relational database (e.g. 126) or both. Each signature schema 
incorporates an understanding of a particular web site's data including relationships 
among the various data (e.g. among its primary informational content found in the body 
of its web pages as well as among such content and associated navigational data (e.g. 
web page links) that govern the data in the page. As described further herein below, 
prior knowledge of the web page code including specific identifiers, tags and text (i.e. 
strings) used within the code (sometimes referred to as "signatures" herein), may be 
used to define instructions to identify portions of the code of interest and to extract 
specific desired data. 

[0039] A signature schema document may be defined for all the pages of a particular 
web site. Large data-driven web sites (e.g. 104) don't maintain thousands of individual 
web pages per se. The sites adopt a few page family templates 108 and dynamically 
populate these with pertinent content from database 112 comprising information (e.g. 
weather, stock data, news, shopping/product data, patent data, trade-mark data etc.) as 
applicable when a client requests a particular page. Each template represents a family 
of pages having objects and attributes. Below are representative example page family 
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templates and their objects and attributes for a web site offering news and an e- 
commerce web site offering products for sale electronically: 

Example 1 : News site 
Family: List Page 

Objects: lists a selection of news stories 
Attributes: Title, abstract and date 
Family: Detail page 

Objects: lists a single news story (and optionally other related stories) 
Attributes: Journalist, City, Date, Title, Full Story, Image 

Example 2: E-commerce site 
Family: List Page 

Objects: lists a selection of products 
Attributes: Image, Item Name, Price, Sale Price 
Family: Search Page (a specific kind of list page) 
Objects: Similar to a list page 
Attributes: Similar to a list page 

[0040] Each family of pages (the family template) can be identified by a "signature" or 
unique set of one or more features that automatically identifies a given page on a web 
site as part of the family and differentiates that family from another family of pages. 
Similarly each object and attribute field of interest can be identified with its respective 
unique signature within a family of pages. A signature schema document typically 
comprise numerous pieces of information (commands), for example, information that 
instructs the engine 140 for: 

identifying all page families; 

identifying and extracting data (i.e. desired objects and attributes) for each page 
family; 

capturing the (implicit or explicit) relationships between the objects and attributes; 
and 
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transcoding the data. 

[0041] A signature schema document may also be configured to enable special 
functionality for the target web site including searching, logging in a user, purchasing 
items, etc. 

[0042] In accordance with a present embodiment, the structure and syntax of a 
representative signature schema document for a representative e-commerce site 
eshop.ca is shown and described. Engine 140 may be configured to receive web page 
code comprising text data and search through the text in accordance with the schema 
document instructions that provide SQL-query like language instructions. Engine 140 
maintains a pointer within the text as it moves through the web page code performing 
various actions, as described below, in accordance with the schema instructions. Table 
1 illustrates a snippet of a representative signature schema: 

1 <?xml version^' 1.0" encodings ISO-8859-1" ?> 

2 <site> 



3 <version major= ,, 1 M minor="27> 

4 <url location="http://www.eshop.ca" key="eshop.ca" name="E-Shop u /> 

5 <advanced> 
6 

7 <indexjink value= ,, http://www.eshop.ca/home.asp ,, /> 

8 </advanced> 

9 <page_type> 

1 0 <lookup type="pex" action= ,, locate_string" name="list_elements n 
id="mylisM" 

ref="Compare products" alt1="Sort products" /> 

1 1 <lookup type="pex" action="locate_string tt name="item_elements" 
id="myitem_1 " ref=""product-details"" /> 

12 <lookup type="pex" action="locate_string" name="menu_elements M 
id=: ,, mymenu_2 ,, ref="anc-lhsnav-subltem H /> 

1 3 <lookup type="pex" action= H locate_string M name="menu_elements n 
id="mymenu_1 " ref= M product-table" /> 

14 <lookup type="pex" action="locate_string M name="item_elements" 
id="myitem^1 H ref= M *'7> 

15 </page_type> 

1 6 <list_elements id="mylist_1 "> 

17 </list_elements> 
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18 <item_elements id="myitem_1"> 

19 <actions> 

20 <lookup type="pex" action= u move_ptr" ref="</head>" /> 

21 </actions> 

22 <element> 

23 <lookup type="pex" action="get_string" name="image" 
ref="largeimageref" location="after" start="<img 
src="" end=""" /> 

24 <lookup type="pex" action="get_string" name="title" ref="product- 
details-prd-title" location="after" start="<span" 
end="</span>" include_sz="1 " strip_tags="1 " /> 

25 <lookup type="pex" action="get_string" name="price" ref="our 
price:" • location="after" start="<td" 
end="</td>" include_sz=" 1 " strip_tags=" 1 " /> 

26 <lookup type="pex" action="get_string" name="sale_price" 
ref="sale price:" location="after" start="<td" 
end="</td>" include_sz=" 1 " strip_tags=" 1 " 
tolerance="1" /> 

27 <lookup type="pex" action="get_string" name="description" 
ref="detailbox-text" location="middle" start="<p" 
end="</p> " include_sz=" 1 " strip_tags=" 1 " /> 

28 </element> 

29 </item_elements> 



Table 1 - XML Signature Schema Snippet for E-Shop.ca 

[0043] In the XML code snippet of Table 1 , instructions at line 4 are for verifying that 
the web page under consideration and the signature schema relate to the same web 
site/domain - eshop.ca. Instructions at lines 9-15 are for determining the particular page 
family to which the web page under consideration belongs. A respective signature that 
defines the particular page family has been previously identified for use to distinguish 
the page. The engine 140 processes the <page type> tag by registering the 
identification strings for each page family. When a web page is obtained by the engine 
as input, the engine may be able to identify the page family by its unique string ref=" and 
the command provides the related tag within the signature schema document where 
further instructions for the particular web pages are found: 

action="locate_string": command to check for the existence of a string. 

name=": identifies the type of page family for each identified family. 
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id=": assigns an id to the page family that is used across the signature schema 
document. 

[0044] For example, at line 10, the instructions identify a web page using the 
alternative signatures "Compare products" or "Sort Products". Web pages with these 
strings are of the same family type. The instructions at line 10 provide a reference tag 
to further instructions for this family, providing a link to instructions for the list_elements 
page family with and ID of mylisM (see lines 16-17). Similarly the other lookup 
instructions provide references to the specific instructions within the signature schema 
document for handling a web page of each web page family. Representative 
instructions for some of the web page families are provided in Table 1 , for example, at 
lines 16-17 and 18-29 with others omitted for brevity. 

[0045] With reference to the extraction instructions for one of the web page families 
(i.e. item_elements id="myitem_1") at lines 18-29, the instruction at line 20 advances 
the scan pointer within the text file of the web page code to a beginning limit of a region 
of interest indicated by a signature reference. This establishes an upper limit for review 
within the text file. Though not shown in this table, an end limit may be defined as well 
(See Table 4). Further such instructions at lines 22-28 may comprise commands to 
locate desired data using "signatures" such as string identifiers that uniquely identify the 
data within the region of interest. In the present example the instructions locate and 
extract one or more elements, namely, product image, title, price, sale price and 
description for a product of the item web page family. For example, instructions at line 
23 extract a string in between the first "<img src="" and """ that appears 
after next appearance of "largeimageref". The string returned is the path (relative URL at 
web site eshop.ca) to the product image. By advancing a search scan pointer within the 
web code to a desired location, references before that location can be skipped when 
searching. Any prior instances of a signature string such as "largeimageref may be 
ignored. In this way, otherwise ambiguous signature references can be avoided. 

[0046] The example in Table 1 shows at least some of the instructions (e.g. lines 23 - 
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27) including one or more directional references relative to the signatures to locate and 
extract the desired data. For example, directional references such as "before" or "after" 
command the engine to extract desired data that is in a relative position in the web page 
before or after the signature string (i.e. ref=). Moreover, such instructions may further 
include at least one of a start reference or an end reference further pinpointing the 
location of the desired data in accordance with that direction. Additional directional 
reference information is discussed herein with reference to code snippets in other 
Tables and the discussion of an embodiment of signature transcoding engine syntax 
presented below. 

[0047] The example within Table 1 demonstrates the extraction of data and the 
establishment of relationships between objects and elements within a same page of a 
web site. However, signature schema documents may further capture relevant 
attributes of an object across pages. For example, a user of client machine 102A may 
click through a number of web pages in eshop.ca to get to a specific product page (e.g. 
Department -> Product Category -> Product Sub-Category -> Specific Product, such as 
TV & Video > 19"-21" TVs > LCD TVs > BrandX Product. The navigational hierarchy 
representing a categorization may be captured and associated to the extracted objects 
and there elements. 

[0048] For brevity, certain instructions were omitted from Table 1 . Tables 2-4 provide 
representative instructions for further web page families for e-shop.ca that may be read 
with Table 1. Table 2 below provides representative instructions, e.g. for lines 16 and 17 
of Table 1 , including instructions for a web page family related to a list of items/products 
for sale. Whereas instructions at lines 22-28 provided product data extraction 
instructions for a web page family showing a single item (i.e. product), the instructions of 
Table 2 provide additional instructions that repeat product data extractions for each 
product in the list. 

1 <list_elements id="mylisM M > " 

2 <paging> 

3 <page_variable value= 'page" /> 
4 <page_start value="0" /> 
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5 <lookup type="pex n action^'geLstring" name^'link" 
ref="Next&nbsp" location="before" start= a <a 
class=" end="</a> B include_sz="1 " strip Jags= u 1 " /> 

6 </paging> 

7 <actions> 

8 <lookup type="pex" action= ,, move_ptr" ref="Sort or compare 
products" ref_alt_1 ="Sort products" /> 

9 </actions> 

1 0 <element> 

1 1 <lookup type="pex" action="get_string" name="link u ref= ,, thumbnail M 

location="before" start="<ahref="" 

end=""> n /> 

1 2 <lookup type="pex M action^'geLstring" name^'image" 
ref="thumbnail" location="middle" start=""" 
end="" n /> 

13 <lookup type="pex" action="get_string" name="title ,, 

ref= ,, class="tx-strong-dgrey&quot;" 
location= B after" start="<a href=" 

end="</a>" include_sz="1" strip_tags="1" /> 

14 <lookup type="pex" action^'^eLstring" name="price ,, ref= n pricepill/" 

location= ,, after ,, start=7" repeat_start= ,, 1 " 

end=".gif tolerance^'l" /> 

1 5 <lookup type="pex M action= ,, move_ptr" ref= M pricepi!l/ ,, /> 

16 </element> 

17 </lisLelements> 



Table 2 - XML Signature Schema Snippet for Product List Web Page Family of E- 

Shop.ca 

[0049] If the engine 140 identifies that the page is of the "mylisM" family, the engine 
determines the location in the signature schema document that contains the signature 
for the objects and elements of that family and applies the instructions therefor. A 
product list at e-shop.ca may span multiple web pages. Instructions at lines 2-6 of Table 
2 find the number of pages and generate the links for each of the pages. Instructions at 
lines 7-9 (action tag ) advance the search scan pointer to the region of web page code 
that may be of interest (i.e. in this case, the start of the list). In this way, a local signature 
reference can be used and any earlier ambiguous references skipped. Skipping to the 
local region of interest may also make the specification of the signature reference less 
complicated. 

[0050] Taking advantage of inherent repeated patterns in the web page code, 
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instructions at lines 10-16 (elements tag) of Table 2 provide product data extraction 
instructions that may be repeated for each product in the list. The engine 140 may be 
provided with commands to scan for each data element of interest using a signature 
reference e.g. ref=", an action, one or more positional instruction(s) to further identify the 
data within the text of the web page code, and any additional text data manipulation 
instructions to extract the desired data (e.g. to remove HTML formatting characters or to 
add characters). The instruction at line 15 moves the scan pointer to the end of the 
object (in this example a product in a list of products) to ready the instructions for 
application against the next object (product) in the list. 

[0051] More particularly: 

lookup type="pex": string lookup 

action ="get_string": returns a value back that is the desired element of the 
object. 

name="link": the object element, in this case the link to the product page 
ref="thumbnail": the reference string that identifies where to find the value of the 
link 

location="before": the value of the link is before the ref string 
start="<a href="": look for the ref string after this value 
end="">": look for the ref string before this value. 



1 <search_elements id="mysearch_1 "> 

2 <settings> 

3 <search_path value="http://www.eshop.ca/search/search.asp" /> 

4 <search_variable value=" keyword" /> 

5 </settings> 

6 <paging> 

7 <page_variable value="page" /> 

8 <page_start value="0"/> 

9 <lookup type="pex" action^'geLstring" name="link" ref="Next&nbsp" 

iocation="before" start="<a href=" repeat start='T 

end="</a>" 

include_sz="1" strip_tags="1" /> 

10 </paging> 

1 1 <actions> 
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12 <lookup type="pex" action="move_ptr" ref="bg-compare-hero" /> 

13 </actions> 

14 <element> 

15 <lookup type="pex" action="get_string" name="link" ref="> u 
location="after" 

start="<a href="" end="">" /> 

16 <lookup type="pex" action="get_string" name="image" ref="<a href 
location="after" 

start="<img src="" end="" M /> 

1 7 <lookup type="pex" action="get_string" name="title" 

ref="class="tx-strong-dgrey&quot; n location="after" 

start="<a 

href=" end="</a>" include_sz="1" strip_tags="1" /> 

1 8 <lookup type="pex" action="move_ptr" ref="bg-compare-hero" /> 

19 </element> 

20 </search_elements> 



Table 3 - E-Shop Search Family Signature Schema Snippet 

[0052] If the engine 140 has identified that the page is of the "mysearch_1" family the 
engine applies the portion of the signature schema document that contains the 
signature for the objects and elements of that family, shown above in Table 3. 

<settings>...</settings>: Contains any web page specific manual overrides such as 
excluding certain menu items, customization, modification of a menu that may be 
desired. In this example, as per line 3 a value of form variable "keyword" will be posted 
to "http://www.eshop.ca/search/search.asp". 



<paging>...</paging>: Manages paging for the search pages. 



<actions>...</actions>: Instruct the engine to move the scan pointer to the string "bg- 
compare-hero" (line 12 of Table 3) and start looking for elements from there. 



<element>...</element>: Contains lookup instructions for each object element as 
previously described. 



1 <menu_elements id="mymenu_1 "> 
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2 <settings> 

3 <blackjist value="Site lndex##External Link" /> 

4 </settings> 

5 <actions> 

6 <lookup type 

7 <Iookup type 

8 </actions> 

9 <element> 

10 <lookuptype 
location= ,, after ,, 

start="<a href=" u end="" n /> 

1 1 <lookup type= u pex" action="get_string n name="title" ref= ,, <li>" 
location=°after" 

start="<a href="" end="</a>" include_sz= n 1" 

stripJags^'T' /> 

1 2 <lookup type="pex" action^' move_ptr M ref= ,, </li>7> 

13 </element> 

14 </menu_elements> 



Table 4 - E-shop Menu Family Signature Schema Snippet 

[0053] If the engine 140 has identified that it is looking for a menu on a page that 
contains the menu style of the "mymenu_1" family, the engine applies the portion of the 
signature schema document that contains the signature for the objects and elements of 
that family, shown above in Table 4. 

<settings>...</settings>: Contains any page specific manual overrides such as 
exclude list, customization, modification, personalization, etc. In this example, as per 
line 3, any result that matches "Site Index", "External Link" are excluded but partial 
matches are also possible by using wild card strings. 

<action>...</action>: Lines 6 - 7 of Table 4 sets the start and end limits to instruct the 
engine 140 where to look for menu items. 

<element>...</element>: Contains lookup instructions for each object element as 
previously described. In this example, lines 10 and 11 of Table 4, an element in 
'mymenu_1' (each individual menu entry of web page) contains link and title as its 
properties. Line 12 instructs the engine to move the pointer to "</li>" to get ready to 
loop through and extract the next menu item with the same elements, taking advantage 
of the repeated patterns within the text of the web page code. 



= a pex w actiQn="move^ptr a ref="bg-lhsnav-title" /> 
= n pex" action="end_ptr" ref= u </table>" /> 

= u pex tt action^geLstring" name= H link u ref="<li> u 
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[0054] Though the example described relates to extracting informational content for an 
e-commerce oriented site, no limitation should be applied. Similar instructions may be 
defined for other types of sites, for pages which permit a user to input information and 
for navigational data extraction. 

[0055] Signature schema document 122 may further comprise transcoding instructions 
(not shown) for use by engine 140 to express the extracted desired data (which may be 
retrieved from database 126) in a target format (e.g. a format of HTML, XML, script etc.) 
for use by the requesting client machine 102. For example, the transcoding instructions 
may define a web page for displaying the extracted data in browser application 86 that 
is suitable for display on the client machine 102. The formatting rules can be system 
and/or user defined and can include one or more parameters such as but not limited to: 
object positioning, object colour, object size, object shape, object font/image 
characteristics, background style, and navigational item display (e.g. in a menu as 
described above) or for display with the content in the generated page on the client 
screen. Browser application 86 (e.g. of machine 102A) may be configured for using a 
markup language (e.g. cHTML) or other code format that is not identical to the code 
provided by web page 110. Alternatively, transcoding instructions may be defined to 
express the extracted desired data in XML or another code format such as for use by a 
different client application or plug-in to a client application such as menu application 82 
or another application (not shown) on client machine 102. 

[0056] Signature schema documents may be prepared (i.e. coded) using a computing 
device such as computing device 128. Computing device 128 may be any suitable 
desktop or laptop device capable of coding documents (which may be but need not be 
XML-type documents) and may be configured to automate or semi-automate coding of 
such documents. 

[0057] Computing device 128 may be coupled to web site 104 to retrieve web pages 
from the site for reviewing to prepare the custom signature schema document for the 
site. Computing device 128 may be configured to automatically review the web page 
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code and apply heuristics or other techniques (e.g. spatial analysis) to determine 
probable content of interest (i.e. desired data) and generate code to extract the desired 
data. For example, primary content of interest tends to be located toward the centre of 
the web page. In another embodiment, the computing device may facilitate a user 
coding signature schema to manually assist with the analysis of the web page and 
identification of desired data and the generation of the instructions. Computing device 
128 may be further coupled to repository 124 to provide (e.g. up-load or publish) coded 
signature schema documents for use by server 120. 

[0058] It will be apparent to a person of ordinary skill in the art that as a web site may be 
re-designed or otherwise changed such that the code of one or more web page families 
may be changed or a family added, an existing signature schema may require re-coding 
to account for the change/addition, as applicable. 

Signature (Transcoding) Engine Syntax 

[0059] In accordance with a present embodiment, further details concerning the syntax 
of schema instructions are described. 

Lookup Syntax 

The lookup tag instructs the engine 140 to perform an insert, delete or query the 
document contents. 

Type: Defines the data type of the lookup. Type may be "pex" for a string 
expression. Type may also support more advanced options such as regular 
expressions, API calls, and SQL queries. 

Action: 

Action = 1ocate_string": Look for a string ("ref identifier") value within the data. Return 
true iff the string exists in the data (i.e. the "ref identifier index >= 0). 

Action = "replace_string": Replace a string within the data with the "ref identifier. 
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Action = "move_ptr": Remove all characters in the data that exist before the location of 
the "ref identifier. 

Action = "end_ptr": Remove all characters in the data that exist after the location of the 
"ref identifier. 

Action = "get_string" Extract a string based on the location of the "ref, "start" , and "end" 
identifiers. 

ID: ID is an identifier of another section within the signature. It allows the result of a 
query to trigger another set of actions within the signature. This is primarily used when 
identifying page types. Once a match has been made, specific instructions are 
executed that are marked with this ID. Recursive data structures (e.g. lists within lists) 
may also be supported. 

Ref: Ref defines the initial identifier that the lookup searches for. If an AND case is 
required multiple ref identifiers can be used (i.e. ref="string1 B ref 1 ="string2"). If an OR 
case is required refjref identifier] _alt_1 can be used (i.e. ref="string1" 
ref_alt_1="string2"). To demonstrate (X='T || Y="2") && (A="8" || B="9") would translate 
to ref="1" ref_alt_1="2" ref1="8" ref1_alt_1="9". 

Repeat_[identifier]: Repeat executes the identifier query additional times. For 
example, if ref="hello" to set the identifier index at the second occurrence of hello the 
following tag would be added: repeat_ref="1". 

Location: 

Location = "before": Search the data in a reverse direction, starting from the "ref 
identifier. This implies that both the "start" and "end" identifier indexes must be less than 
the "ref index. 

Location = "middle": Search the data in two directions, starting from the "ref identifier. 
This implies that the "ref" identifier index is greater than the "start" identifier index and 
less than the "end" identifier index. 
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Location = "after": Search the data in a forward direction, starting from the "ref" identifier. 
This implies that both the "start" and "end" identifier indexes must be greater than the 
"ref index. 

Start: Start is primarily used when action="get_string" and may also be used for 
replace/remove instructions. . The start identifier index will be the start index of the 
string to extract. If an AND case is required multiple "start" identifiers can be used (i.e. 
start="string1" startl ="string2"). If an OR case is required startjstart identifier] _alt_1 
can be used (i.e. start="string1" start_alt_1 ="string2"). To demonstrate (X="1" || Y="2") 
&& (A="8" || B="9") would translate to start="1" start_alt_1="2" startl ="8" 
startl _alt_1 ="9". To find the n m match see the repeat syntax. 

End: End is primarily used when action="get_string" and may also be used for 
replace/remove instructions. If an AND case is required multiple "end" identifiers can 
be used (i.e. end="string1" end1="string2"). If an OR case is required end_[end 
identifier] _alt_1 can be used (i.e. end="string1" end_alt_1="string2"). To demonstrate 
(X="1" || Y="2") && (A="8" || B="9") would translate to end="1" end_alt_1="2" end1="8" 
end1_alt_1="9". To find the n ,h match see the repeat syntax 

Maxjndex: Maxjndex is used to limit the scope of a query by ensuring that no other 
identifier index is greater than the "maxjndex". . If an AND case is required multiple 
"maxjndex" identifiers can be used (i.e. max_index="string1" maxjndex1="string2"). If 
an OR case is required max_index_[ maxjndex identifier] _alt_1 can be used (i.e. 
maxjndex="string1" maxjndex_alt_1="string2"). To demonstrate (X="1" || Y="2") && 
(A="8" || B="9") would translate to max_index="1" maxjndex alt_1="2" maxjndex ="8" 
maxjndex _alt_1="9". To find the n" 1 match see the repeat syntax. 

MaxJndex_Use_Ref: Max_lndex_Use_Ref is a Boolean value set to 0 or 1 . It is used 
with Maxjndex. When set to 0, the "maxjndex" will begin querying at the beginning of 
the data. When set to 1 , the "maxjndex" will begin querying from the "ref identifier 
index. 
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Gbl_append_[identifier]: GbLappend appends a string passed via the url to the 
identifiers query value 

Gbl_Repeat_[identifier]: GbLRepeat executes the identifier query additional times. 
For example, if ref="hello" to set the identifier index at the second occurrence of hello 
the following tag would be added: gbl_repeat_ref="var" where var would be passed in 
the URL i.e. http://www.eshop.ca/mobile/fatfree.asp?site=...&url=...&var=1. 

Tolerance: Tolerance is a Boolean value set to 0 or 1 . It is used to return an empty 
string. By default tolerance is set to 0 which enforces that a property be found on a 
page, otherwise the page will be marked as "invalid" and an appropriate error message 
returned. When set to one, an empty value is returned for properties that can not be 
located. 

Include.sz: lnclude_sz is a Boolean value set to 0 or 1 and used with get.string. It is 
by default set to 0. When set to 1 it includes the "start" value and the "end" value as 
part of the result. 

Include_start: lnclude_start is a Boolean value set to 0 or 1 and used with get_string. 
It is by default set to 0. When set to 1 it includes the "start" value as part of the result. 

Include_end: lnclude_end is a Boolean value set to 0 or 1 and used with get_string. It 
is by default set to 0. When set to 1 it includes the "end" value as part of the result. 

Closetag: Closetag is a Boolean value set to 0 or 1 and used when action="get_string". 
It appends /> to the extracted value. 

Strip_Tags: Strip_Tags removes HTML tags from the value and used when 
action="get_string". 

Strip_tags="1": remove all tags. 

Strip_tags="2": remove all br and script tags. 
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Strip_tags="3": remove all tags except replace </p> </li> with <br>. 

Strip_tags="4": remove all tags except replace </div> <br> with <br>. 

Strip_tags="tag1,tag2...tagN n :remove all tag1, tag2,...tagN leaving any tag not listed. 

Notrlm: Notrim is a Boolean value set to 0 or 1 and used when action="get_string H . By 
default all value have white spaced trimmed. When this property is set to 1 , white space 
is not trimmed. 

Append: Append is a string value and used when action="get_string". It appends a 
string to the extracted value. 

Prepend: Prepend is a string value and used when action^get.string". It prepends a 
string to the extracted value. 

Upper: Upper is a Boolean value set to 0 or 1 and used when action="get_string". It 
converts all characters to upper case. 

Lower: Lower is a Boolean value set to 0 or 1 and used when action="get_string". It 
converts all characters to lower case. 

Page Syntax 

The page syntax extracts the paging information from the data. This allows the end 
user the ability to change pages just as on the desktop. 

Page_variable: Defines unique key that defines a family's paging feature. 

Page_start: Defines value of first page in a family's paging feature. 

Page_post: Path where paging variable(s) must be transmitted to. 

Page_start : Defines value of first page in a family's paging feature. 

Page ^increment: Defines value that paging increases by for each page in a family's 
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paging feature. 

Page_block: Defines unique key that defines a family's paging block feature. 

Page_block_size: Defines the size of the family's page block, (i.e. 10 items per page) 

UrLappend: Append the unique key that defines a family's paging feature and the page 
number. 

Search Syntax 

Make a website family's search feature functional by specifying details such as what 
variable to post. 

Search_path: Search path where search variable must be transmitted to 

Search_variable: Name of search variable which a website's search feature is looking 
to read, request, post, etc. 

UrLreplace: Remove a portion of the url that is specific to posting search parameters 
URL Syntax 

The url tag defines global properties for a site, including the url, and name: 
<uri location="http://www.eshop.ca" key="eshop.ca" name="E-Shop" /> 
Name: Name is the name to display when browsing using the gateway 120 
Location: Location defines the fully qualified address of the site. 
Key: Key is the site. 
Advanced Syntax 

The advanced tag defines global properties for the site. This at a minimum includes the 
path to the initial page of the site. 
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<advanced> 

<index_link value="http://www.eshop.ca" /> 

<check_out value="1" /> 
</advanced> 

Indexjink: Indexjink specifies the path to the initial page of the site. This is usually 
the same page as the location property from the URL syntax. This field is always 
required. 

Appendjink: Appends a string value to every URL requested for this site. 

No_purchase: No_purchase is a Boolean value 0 or 1 . The default value is 0 which 
implies that an item should contain a purchase link. When true, the purchase link is 
removed. 

No_item: Nojtem is a Boolean value 0 or 1 . The default value is 0 which implies that 
Item pages should show up in the breadcrumb. When true, the item is not added to the 
breadcrumb. 

Check.out: Check_out is a Boolean value 0 or 1 . The default value is 0 which implies 
that Item purchase link sends the request and control away from the gateway server 
120. When true, then a checkout process has been created for use with gateway server 
120. 

Product_img_width: Product_img_width defines the width of all item images. 

Use.cookies: Use_cookies a Boolean value 0 or 1 . By default it is set to 0, and 
cookies are not passed to the site. When true, gateway 120 passes all cookies from 
client machine 102 to the site 104, and from the site 104 to the client machine. 

Page Type Syntax 

The page type is a collection of lookup queries that have an id associated with them. 
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Lookup queries may be processed in a top down fashion. The first successful lookup 
will trigger another section in the signature schema document. For example, if the 
following evaluates to true: 

<page_type> 

<lookup type="pex" action="locate_string" name="list_elements" id="mylist_1 " ref="<l- 
-"/> 

</page_type> 

Then the tag element <list_elements id= "mylisM "> would be executed next. 
General Element Syntax 

Elements include list_elements, menu_elements, item_elements, search_elements, 
form_elements. Each element has an ID. For example a menu element: 

<menu_element id="menu_id"/> 

The element may contain the following sub containers (settings, actions, elements, 
paging) which scope resides only within the element. Each element is associated with a 
specific rendering function. 

<menu_element id="menu_id7> 

<settings> </settings> 

<paging> <l paging > 

<elements> <J elements > 
<actions> <l actions > 
</menu_element> 

Settings Syntax 

Settings syntax varies based on the type of element it resides in. Settings allow 
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customizations that only apply to a specific page family. 

Blackjist - menu_elements: Blackjist removes menu items with names that reside 
in the black list. Each entry is separated delimited (e.g. using two pound characters 
(##). 

Passjmage - list elements, search_elements: Pass_image adds the image path to 
the url when requesting an item. The image added to the url will be used as the item 
image. 

Pricefn] - item ^elements: Price[n] where n is an integer renames the rendered item 
with name price[n]. 

Action - form_elements: Overrides the action of a form displayed to the end user. 
Handle - form_elements 

Handle = "display" - display the form to the end user. 
Handle = "post" - post the form. 
Handle = "get" - get the form. 

Cookie - form_elements: Send additional cookies when posting this form. 

Input_[identifier] - form.elements: Input tag adds/modifies a form value with name 
[identifier] setting its value. 

RenameJIdentifier] - form_elements: Rename tag renames a form value with name 
[identifier]. 

Actions Syntax 

The actions tag primary function is data manipulation. It contains lookup queries that 
modify data with actions of "move_ptr" or "end_ptr". 
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<actions> 

<lookup type="pex n action= n move_ptr" ref= u </head> a /> 
</actions> 

[0060] Persons of ordinary skill in the art will appreciate that alternative embodiments 
are contemplated. System 100 may be implemented so that one or more web sites 
103, 104 and 105 are coupled via to the telecommunication network (either alone by a 
server 106 or by one or more web servers like web-server 106), and that a 
corresponding plurality of schemas for each of those web sites (or each of the web 
pages therein, or both) can be maintained by gateway and schema server 120 and 
repository 124. Client machines 102 can be configured for proxied connections through 
different servers 120 and for accessing aggregated web site data from database 126. 
Those skilled in the art will now further recognize that servers 120 and web server 125 
can be hosted by a variety of different parties, including, for example but without 
limitation: a manufacturer of client machine 102, a service provider that provides access 
to the telecommunication network on behalf of user U of a client machine 102; the entity 
that hosts web-site 104 or a third party intermediary. In web site host example it can 
even be desired to simply combine the web server 106 and schema server engine 120 
on a single server to thereby obviate the need for separate servers. Alternatively the 
functionality of server 120 and web server 125 may be locally resident on the client 
machine providing. 

[0061] Figure 4 is a schematic representation of a aggregate website search database 
126. The database 126 contains one or more tables for storing aggregate web page 
data extracted from target web sites. The database may be a relational database 
enabling structured database queries. For an e-commerce web sites the database 126 
may contain tables defining a web site identification and category index 402 and 
category data 404 containing specific item details, as will be discussed in connection 
with Fig. 5. The category data 404 may be further broken down into product data 406 
table if further data is required. Indexes 408 can be created to reference aggregate web 
site data to improve query responsiveness and results. Metadata 410 associated with 
the original web pages may be stored such as images, web page formatting or 
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navigation data. In other web site applications such as news, weather, stock data, 
patent data, trade-mark data etc., the appropriate tables would be configured to store 
the extracted data based upon the signature scheme. 

[0062] Figure 5 is a schematic representation of tables in an aggregate website search 
database for e-commerce. The extracted data from selected e-commerce websites can 
be formatted into category index 402 which identifies 502 the website where the data 
was extracted from and the category 504 of interest. In this example entry 506 
identifies http://www.eshop2.ca/pg=3 as the reference for where digital cameras can be 
indexed. For each category in the category index 402, an individual category 404 table 
can be created. The category table contains location identifiers 510 for each product 
retrieved from the web site based upon the defined signature schema and populated at 
step 31 0 during the retrieval of data from the web site (or domain). The vendor 51 2, title 
514 of the product, price 516 and description 518 extracted can then be stored. It 
should be understood that the categories identified can be tailored to the application. 
The database then provides a means for querying the aggregate data from the web site 
to present meaningful information to the client machine 102A. By storing the aggregate 
data queries can be created to meet desired requirements and transcoded for 
presentation on the client machine 102A. 

[0063] Figure 6 is a schematic representation of a index table 408. The table may be 
configured by a database management system (DBMS) based upon common queries 
generated to the aggregate database. The indexes enable improved queries by cross- 
indexing data contained in extracted web page data tables 402, 404 and 406 . For 
example, an index may be define by categories 610 against suppliers 612. At entry 622 
laptops from Vendor A is available at three suppliers. A query to the database 126 can 
then be generated by engine 140 to determine which vendor has the lowest price. 
Additional indexes may be defined such as a geographic index to identify locations of 
products or vendors. The index may be more specific such as defining sub categories 
of products or versions. 

[0064] Figure 7 illustrates a method of creating the aggregate website search database 
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smart indexes for searching. The web sites 103, 104 and 105 that are to be indexed 
are identified, and page request(s) 702 from data collection engine 152 (also known as 
a bot or web crawler) hosted on web server 125. The data collection engine 150 is a 
web crawler to independently collect data from target web sites for storage in aggregate 
database 126. Alternatively, the data collection engine 150 may be included in the 
engine 140 of gateway 120. The web site data may be retrieved or indexing periodically 
by data collection engine 152 which may be triggered manually, scheduled or on-going. 
The signature schema for the web site is retrieved 704 from repository 124 by engine 
140. The schema is applied 706 to the received web site data to extract relevant data 
from the web page content. An index can then be generated 708 based upon the 
defined signature schema structure to index the stored data more efficiently. As the 
schema is defined to extract elements with which objects and their attributes on the web 
page can be defined or described and the schema incorporates knowledge of what 
these objects and attributes represent, an intelligent and indexed database 126 can be 
defined. The extracted data and the generated index is stored 710 to the aggregate 
database 126 in the appropriate tables. The process is repeated for each relevant web 
site to generate the aggregate web site search database. The data collection engine 
150 can then periodically access web sites 103, 104, 105 to ensure stored data and 
indexes are accurate and up to date. The generated indexes provide a mechanism to 
search websites for specific content, as opposed to keywords based searches. The 
indexes are generated based upon data extracted from the target web sites, in contrast, 
traditional solutions require data to be submitted to enable searches. 

[0065] Figure 8 illustrates a method of querying the aggregate website search database 
utilizing smart indexes for searching. A search is generated by a user via an interface 
on client machine 102A and received 802 at gateway 120. The search can be based on 
any parameters defined in the database that and extracted by the signature scheme as 
opposed to keyword only queries. Knowledge based queries can be generated 
providing more relevant search results using the aggregate database content. Engine 
140 processes the query and determines 804 the appropriate database index 408. A 
relevant database query such as SQL (Structured Query Language) using an 
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associated index 408 is generated 806. The query is executed against the database 
data and retrieved 808 from the database. The appropriate web data can then be 
provided 810 to the client machine 102A. Optionally, the retrieved data can be 
transcoded using a signature schema (which may be relevant to the original web page 
content) to accommodate client machine 102A characteristics. For example as shown 
in Fig 7C. 
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CLAIMS 

1 . A method for generating an aggregate website search database index, the method 
comprising: 

sending a page request to a web site selected from one or more web sites; 
receiving the requested web page from the selected web site; 
retrieving signature schema associated with the selected web site; 
applying signature schema to the requested web page to extract data identified 
by the signature schema; 

creating an index using the signature schema defined for the extracted data for 
the one or more web sites; and 

storing extracted data and indexes to an aggregate database comprising data 
extracted from the one or more web sites wherein the indexes define 
relationships between data extracted from the requested web page identified by 
the signature schema. 

2. The method of claim 1 further comprising: 

receiving a search query from a client machine for data stored in the aggregate 
database; 

determining a database index associated with one or more parameters of the 
search query; 

generating a database query based upon the received search query using the 
determined index; and 

retrieving data from the aggregate database based on the database query. 

3. The method of claim 2 further comprising: 

generating a retrieved data web page for the client machine using the retrieved 
data; and 
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providing the retrieved data web page to the client machine. 

4. The method of claim 3 wherein the client machine hosts a push based application for 
receiving the requested web page data. 

5. The method of claim 1 wherein the database is a relational database. 

6. The method of claim 2 wherein the database query is a Structured Query Language 
(SQL) query. 

7. The method of claim 1 comprising maintaining a repository of signature schemas, 
wherein each signature schema is defined each of the one or more web sites. 

8. The method of claim 6 wherein the signature schema comprises extensible Markup 
Language (XML) documents comprising query language for extracting data from the 
requested web page. 

9. The method of claim 1 wherein the index comprises a category identifier and a web 
site hypertext markup language (HTML) link. 

10. The method of claim 1 further comprising storing the aggregate database and 
indexes on the client computer. 

1 1 .The method of claim 9 wherein the client computer is a wireless device. 

12. A system for generating an aggregate website search database using smart indexes 
for searching, the system comprising: 

at least one computing device comprising a processor and a memory coupled 
thereto, said memory storing instructions and data for configuring the processor 
to provide a engine to: 

send a page request to a web site selected from the one or more web 
sites; 

receive the requested web page from the selected web site; 
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retrieve signature schema associated with the selected web site; 
apply signature schema to the requested web page to extract data 
identified by the signature schema; 

create an index using the signature schema defined for the extracted data 
for the one or more web sites; and 

store extracted data and indexes to an aggregate database comprising 
data extracted from the one or more web sites wherein the indexes define 
relationships between data extracted from the requested web page 
identified by the signature schema. 

13. The system of claim 11 the engine is further configured to: 

receive a search query from a client machine for data stored in the aggregate 
database; 

determine a database index associated with one or more parameters of the 
search query; 

generate a database query based upon the received search query using the 
determined index; and 

retrieve data from the aggregate database based on the database query. 

14. The system of claim 12 further configured to: 

generate a retrieved data web page for the client machine using the retrieved 
data; and 

provide the retrieved data web page to the client machine. 

15. The system of claim 11 wherein the database is a relational database. 

16. The system of claim 14 wherein the database query is an Structured Query 
Language (SQL) query. 

17. The system of claim 9 wherein the signature schema are extensible Markup 
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Language (XML) documents comprises query language for extracting data from the 
requested web page. 

1 8. The system of claim 12 wherein the index comprises a category identifier and a web 
site hypertext markup language (HTML) link. 

19. The system of claim 11 further comprising a repository of signature schemas 
wherein each schema is defined for each of the one ore more web sites. 

20. The system of claim 11 wherein the aggregate database and indexes are stored on 
the client computer. 

21 .The system of claim 20 wherein the client computer is a mobile device. 

22. The system of claim 12 wherein the client machine hosts a push based application 
for receiving the web data. 

23. A computer program product storing computer readable instructions which when 
executed by a computer processor configure the processor for: 

sending a page request to a web site selected from one or more web sites; 
receiving the requested web page from the selected web site; 
retrieving signature schema associated with the selected web site; 
applying signature schema to the requested web page to extract data identified 
by the signature schema; 

creating an index using the signature schema defined for the extracted data for 
the one or more web sites; and 

storing extracted data and indexes to an aggregate database comprising data 
extracted from the one or more web sites wherein the indexes define 
relationships between data extracted from the requested web page identified by 
the signature schema. 
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24. A method for generating an aggregate website search database index, the method 
comprising: 

sending a page request to a web sites selected from the one or more web sites; 

receiving the requested web page from the web site; 

retrieving signature schema associated with the selected web site; 

applying signature schema to the requested web page to extract data identified 

by the signature schema; 

creating an index using the signature schema defined for the extracted data for 
the one or more web sites; 

storing extracted data and indexes to an aggregate database comprising data 
extracted from the one or more web sites wherein the indexes define 
relationships between data extracted from the requested web page identified by 
the signature schema and wherein the signature schema comprise extensible 
Markup Language (XML) documents comprising query language for extracting 
data from the selected web page; 

receiving a search query from a client machine for data stored in the aggregate 
database; 

determining a database index associated with one or more parameters of the 
search query; 

generating a database query based upon the received search query using the 
determined index; 

retrieving data from the aggregate database based on the database query; and 
providing the retrieved data to the client machine. 
25. The method of claim 24 where in the client machine is a wireless device. 
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